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Abstract 

This  is  a  summary  of  a  two-volume  final  report  of  a  contract  to  develop 
measures  of  short-term  outcome  for  eight  common  health  problems. 
The  first  volume  is  a  conceptual  overview  of  the  role  of  outcome  in- 
formation in  quality  assessment,  including  an  extensive  annotated  bib- 
liography. The  second  volume  describes  the  epidemiology  of  each  con- 
dition chosen  for  study  and  the  results  of  an  expert  panel  meeting  at 
which  disease-specific  outcome  measures  were  developed.  The  eight 
problems  covered  are  asthma;  breast  mass  and  breast  cancer;  cholecyst- 
ectomy; diarrhea  and  dehydration  in  children;  ischemic  heart  disease; 
osteoarthrosis;  otitis  media  in  children;  and  tonsillectomy  with  or  with- 
out adenoidectomy.  Summary  tables  of  the  measures  suggested  for 
each  condition  are  included  in  each  chapter.  Also  included  are  meas- 
ures of  risk  factors  which  would  have  to  be  controlled  in  order  to  ad- 
just for  patient  mix  differences  in  interpretation  of  group  outcomes. 
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□  □  D 


Foreword 


One  of  the  major  tasks  of  health  services  research  is  to  develop 
methods  for  monitoring  the  performance  of  the  health  care  delivery 
system.  A  significant  part  of  this  effort  is  directed  at  the  problem  of 
measuring  and  assessing  the  quality  of  care. 

While  this  research  has  a  long  history,  most  attempts  to  measure  qual- 
ity have  focused  either  on  the  structure  or  the  process  of  care,  and  not 
on  health  outcomes.  This  tendency  partly  reflected  the  difficulty  of 
measuring  outcome.  It  also  reflected  a  general  view  that  improved 
health  care  would  lead  to  better  health,  and  that  the  way  to  better  care 
was  the  infusion  of  more  resources.  The  prevailing  mentality  was, 
"When  in  doubt,  do  it." 

Recently,  public  policy  has  taken  a  new  direction.  Concerns  about  ris- 
ing costs,  combined  with  an  increasing  awareness  of  the  limitations  of 
devoting  more  resources  to  health  care  in  order  to  achieve  better 
health,  have  given  rise  to  policies  directed  at  restricting  the  use  of  re- 
sources. The  current  emphasis  on  cost  containment  represents  a  radi- 
cal move  away  from  the  old  perspective,  and  puts  the  burden  of  proof 
on  those  who  would  provide  more  and  costlier  services.  The  new  rule  is 
becoming,  "When  in  doubt,  leave  it  out."  This  change  demands  new 
ways  of  measuring  and  assessing  the  quality  of  medical  care  services. 

What  is  required  is  a  marked  improvement  in  our  ability  to  relate  the 
use  of  health  services,  and  the  resources  they  require,  to  health  out- 
comes. To  be  most  effective,  expenditure  limiting  strategies  should  be 
directed  at  amenity  and  not  necessity.  Although  the  distinction  be- 
tween the  two  has  been  elusive,  the  effect  of  different  therapeutic 
processes  on  outcomes  will  be  the  central  concern  both  in  the  im- 
plementation of  cost  containment  policies  and  in  the  assessment  of 
their  effectiveness. 

The  study  presented  here,  supported  by  the  National  Center  for 
Health  Services  Research,  represents  a  significant  step  in  addressing 
that  issue.  It  provides  a  framework  for  the  continuing  development  of 
outcome  measures  and  offers  several  examples  of  both  the  feasibility 
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and  applicability  of  this  perspective.  Particularly  in  light  of  the  current 
policy  climate,  this  is  an  important  and  timely  advancement  of  the 
health  services  research  effort. 

Gerald  Rosenthal,  Ph.D. 
Director, 

National  Center  for  Health  Services  Research 
August  1977 
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Preface 


The  purpose  of  the  research  was:  (1)  to  prepare  a  conceptual 
paper  that  would  place  the  outcome  method  for  assessing  quality  of 
medical  care  in  perspective,  and  (2)  to  develop  disease-specific  short- 
term  outcome  measures  that  could  be  used  to  evaluate  the  quality  of 
the  medical  and/or  surgical  care  delivered  to  patients  with  one  of  eight 
different  disease  conditions  or  surgical  procedures. 

The  first  volume  of  the  final  report  (PB  258  613)  presents  an 
overview  of  the  outcome  method  of  assessing  the  quality  of  care,  with 
reference  to  theoretical  issues  and  to  the  way  in  which  the  method  has 
been  employed  in  previous  quality  of  care  research.  The  report  details 
the  specific  approach  taken  to  develop  outcome  criteria  for  eight  dis- 
ease conditions,  and  presents  the  outcome  standards  that  are  of  highest 
priority  for  use  in  assessing  quality  of  care  delivered  to  patients  with 
the  conditions  studied.  The  volume  concludes  with  a  series  of  observa- 
tions as  to  the  feasibility  of  the  outcome  method  in  general,  and  as 
applied  to  this  research,  and  offers  suggestions  for  further  develop- 
mental research  on  methods  of  quality  assessment. 

The  second  volume  (PB  258  614)  contains  technical  chapters  on 
each  of  the  eight  conditions  and  procedures  studied:  asthma  in  chil- 
dren and  adults;  breast  mass  and  breast  cancer;  cholecystectomy; 
diarrhea  and  dehydration  in  children;  ischemic  heart  disease  (includ- 
ing angina  pectoris,  myocardial  infarction,  and  sudden  death);  osteoar- 
throsis; otitis  media  in  children;  and  tonsillectomy  with  and  without 
adenoidectomy.  Included  are  the  outcome  criteria  and  standards  pro- 
posed for  use  in  measuring  the  quality  of  care  delivered  to  patients 
with  these  conditions. 

This  NCHSR  Research  Summary  gives  a  brief  executive  summary 
of  the  conceptual  discussion  included  in  the  first  volume,  along  with  a 
description  of  the  specific  method  used  and  an  illustration  of  the  types 
of  outcome  and  standards  developed. 
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Introduction 


The  quality  of  personal  medical  care  services  is  a  matter  of  increas- 
ing public  concern.  Major  efforts  are  under  way  to  increase  the  quality 
of  medical  care  services  through  establishment  of  ongoing  quality  as- 
surance mechanisms.  However,  these  efforts  are  hampered  by  lag  in 
development  of  adequately  tested  methods  for  quality  assessment. 

Quality  of  care  assessments  deal  with  one  or  more  of  the  three 
following  measures  of  medical  care:  structure,  process,  or  outcome. 
Most  of  these  are  based  on  the  common  assumption  that  adequate  re- 
sources (structure)  contribute  to  adequate  diagnostic  workups  and 
treatment  (process),  which  in  turn  result  in  favorable  health  (outcome). 

Within  this  conceptual  framework,  many  studies  have  focused  on 
developing  and  testing  ways  to  measure  the  adequacy  and  quality  of 
the  process  of  medical  care.  There  is,  however,  a  major  problem  with 
measurement  of  quality  through  use  of  process  indicators  alone:  the 
relationship  between  the  medical  care  process  itself  and  health  status  is 
not  always  direct.  In  many  cases,  it  may  be  so  confounded  by  interven- 
ing variables,  such  as  patient  compliance,  that  adequate  process  may 
not  result  in  good  outcomes.  On  the  other  hand,  apparently  poor  proc- 
ess may  result  in  good  outcomes  if  the  process  criteria  selected  are  in- 
valid or  are  incorrectly  measured.  Thus,  the  validity  of  using  process 
measures  alone  as  indicators  of  the  quality  of  medical  care  has  been 
questioned.  Several  studies  also  demonstrate  little  relationship  between 
quality  assessments  made  using  structural  measures  and  those  using 
process  measures.  These  statements  about  process  and  structural 
measures,  however,  must  be  qualified.  Relationships  among  structural, 
process,  and  outcome  measures  in  a  quality  assessment  may  become 
more  apparent  and  the  use  of  structural  and  process  measures  in  as- 
sessing quality  more  appropriate  when  these  terms  are  defined  and 
conceptualized  more  precisely  and  when  they  are  measured  more  reli- 
ably. 

In  contrast,  the  face  validity  of  outcome  assessment  is  more  appar- 
ent, since  measurement  of  outcome  necessitates  measurement  of 
"health"  itself,  or  some  aspect  of  it.  However,  when  outcome  measures 
have  been  proposed  for  use  or  applied  to  evaluate  quality  of  care,  sev- 
eral problems  have  arisen: 

1.  The  outcomes  most  frequently  used,  such  as  death  or  incidence  of 
major  complications,  may  be  so  uncommon  that  detection  of  sig- 
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nificant  differences  between  patient  groups  requires  samples  so 
large  that  the  feasibility  of  the  study  is  limited. 

2.  "Ultimate"  outcomes,  such  as  death  and  restoration  of  normal 
function,  often  occur  so  late  in  the  course  of  treatment  that  timely 
evaluation  is  impossible. 

3.  Such  commonly  used  measures  as  disability  and  return  to  function 
are  heavily  influenced  by  intervening  factors,  such  as  genetic 
makeup  and  physical  and  social  environment,  that  are  beyond  con- 
trol of  the  medical  care  system  proper. 

4.  Information  about  many  outcomes  is  not  readily  available,  requir- 
ing follow-up  interviews.  These  are  expensive  to  conduct  and  dif- 
ficult to  complete  for  a  high  proportion  of  the  relevant  patient 
population. 

5.  Information  is  lacking  on  the  breadth  of  the  outcome  criteria  that 
should  be  used  in  assessing  quality  of  care.  Should  outcome  as- 
sessment be  limited  to  physical  and  physiologic  measures,  or 
should  it  include  psychological  measures  such  as  sexual  function 
following  a  radical  mastectomy  for  breast  cancer? 

The  problems  enumerated  above  lead  at  least  to  questions  about  the 
feasibility,  reliability,  and  validity,  of  the  outcome  method  in  assessing 
quality  of  care.  As  if  these  problems  were  not  enough,  there  are  cur- 
rently no  outcome  criteria  and  standards  available  for  use  by  opera- 
tional quality  assurance  programs.  Despite  this  fact,  the  outcome 
method  is  being  promoted  heavily.  There  is  a  legal  mandate  for  medi- 
cal care  evaluation  studies  to  be  performed  under  the  auspices  of  Pro- 
fessional Standards  Review  Organizations  in  all  hospitals  with  Medicare 
or  Medicaid  patients.  The  Bureau  of  Quality  Assurance,  the  agency  in 
the  Department  of  Health,  Education,  and  Welfare  responsible  for  im- 
plementing the  program,  encourages  use  of  outcome  assessment  when- 
ever possible  in  these  evaluations.  The  Joint  Commission  on  the  Ac- 
creditation of  Hospitals,  a  voluntary  nongovernmental  agency,  has  de- 
veloped a  Performance  Evaluation  Procedure  that  must  be  im- 
plemented by  hospitals  in  order  to  be  reaccredited.  The  procedure  de- 
pends heavily  on  assessment  of  appropriate  patient  outcomes.  Finally, 
the  Health  Maintenance  Organization  (HMO)  Act  of  1973  established  a 
second  federally  mandated  quality  assurance  system.  The  law  requires 
implementation  of  quality  assurance  systems  in  all  federally  supported 
HMOs  to  assess  the  quality  of  care  using  outcome  measures  that  reflect 
both  provider  and  consumer  values.  In  summary,  there  exists  a 
paradoxical  situation,  in  which  policy  demands  that  operational  quality 
assurance  systems  use  the  outcome  method  to  assess  quality  of  care, 
while  there  is  a  dearth  of  valid  and  reliable  outcome  criteria  and  stand- 
ards and  no  method  of  proven  feasibility  by  which  they  can  be  applied. 

In  an  attempt  to  address  some  of  these  problems  and  further  the 
state  of  the  art  of  quality  assessment  using  the  outcome  method,  the 
National  Center  for  Health  Services  Research  contracted  with  The 
Rand  Corporation  to  prepare  a  conceptual  overview  of  the  outcome 
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method  and  to  develop  "proximate"  (short-term)  outcome  measures 
for  use  in  assessing  the  quality  of  care  delivered  to  patients  with  any 
one  of  eight  different  disease  conditions. 

This  research  was  limited  by  the  scope  of  the  contract  insofar  as  it 
was  viewed  as  a  first  phase  or  developmental  effort.  Thus,  many  con- 
ceptual problems  were  not  addressed  in  the  technical  work  done  to 
develop  outcome  criteria  and  standards.  In  particular,  two  major  lim- 
itations were  placed  on  the  research.  It  was  not  possible  to  collect  new 
data  to  test  the  validity  of  the  outcome  criteria  and  standards  estab- 
lished. Moreover,  disease-specific  process  criteria  were  not  developed 
simultaneously,  thereby  negating  comparison  at  this  time  of  the  validity 
of  process  and  outcome  approaches  to  assessing  quality  of  care. 


□  □  □ 


Conceptual  overview: 
the  outcome 
method  of  quality 
assessment 


The  ways  in  which  outcome  information  can  be  used  to  assess  qual- 
ity of  medical  care  can  be  summarized  as  follows: 

1.  To  monitor  prospectively  the  quality  of  care  provided  by  groups 
of  physicians  or  by  individual  providers  as  it  is  delivered,  in  order 
both  to  arrive  at  a  value  judgment  about  quality  and  to  intervene 
and  change  the  care  process  when  necessary  to  avoid  adverse  out- 
comes; and 

2.  To  monitor  retrospectively  the  quality  of  care,  for  one  of  two 
purposes: 

a.  within  a  physician's  office  or  a  single  delivery  system,  to  iden- 
tify problems  in  health  outcomes  that  result  from  inappro- 
priate or  inadequate  processes  of  care  and  to  change  these 
processes  to  achieve  closer-to-optimal  outcomes;  or 

b.  to  do  a  comparative  evaluation  of  different  aspects  of  a  deliv- 
ery system  (whether  a  single  system  or  a  local,  regional,  or  na- 
tional system)  in  order  to  make  value  statements  that  will  sup- 
port policy  decisions. 

Prospective  monitoring  for  quality  of  care  assessment 

While  prospective  monitoring  of  outcomes  is  used  by  every  physi- 
cian to  check  patient  progress,  no  examples  could  be  found  in  the  lit- 
erature of  the  use  of  outcomes  for  the  dual  purposes  of  noting  clinical 
progress  and  assessing  quality.  Increasing  availability  of  populations  of 
known  size  associated  with  specific  medical  care  delivery  systems  (e.g., 
group  practices  and  HMOs)  may  make  possible  the  use  of  such  tech- 
niques. Professional  Standards  Review  Organizations  might  also  use 
prospective  monitoring  techniques  in  performing  medical  care  evalua- 
tion studies. 

Retrospective  monitoring  for  quality  of  care  assessment 

In-house  quality  assessment.  Retrospective  monitoring  of  the  qual- 
ity of  care  is  generally  undertaken  with  one  of  two  purposes  in  mind. 
The  first  is  to  identify  and  correct  major  deficiencies  in  quality  of  care. 
Such  a  study  is  usually  designed  for  use  in  changing  practice  behavior 
in  an  identifiable  medical  care  delivery  system,  such  as  a  hospital, 
clinic,  or  solo  or  group  practice.  Results  are  not  intended  for  publica- 
tion, and  comparisons  with  other  systems  are  not  made. 
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Considerable  effort  has  been  spent  on  developing  quality  of  care  as- 
sessment methods  that  use  outcome  information  for  this  purpose.  De- 
spite the  developmental  work,  none  of  the  approaches  has  been  rigor- 
ously evaluated  as  to  its  ability  to  maximize  the  health  status  of  patients 
in  the  medical  care  sites  or  systems  that  have  instituted  such  quality 
assurance  activities.  At  present,  it  is  perhaps  fair  to  say  that  quality 
assurance  activities  based  on  outcome  information  have  contributed  lit- 
tle to  improvements  in  health. 

Outcome  assessment  for  policymaking.  The  second  purpose  to 
which  retrospective  monitoring  of  quality  of  care  via  outcomes  can  be 
put  is  comparative  evaluation  of  different  aspects  of  a  delivery  system, 
the  results  of  which  are  intended  to  support  policy  decisions.  While 
policy-relevant  quality  assessment  sponsored  by  the  federal  govern- 
ment is  a  major  example  of  this  use  of  outcome  information,  it  can  also 
be  undertaken  at  a  regional  level  or  within  a  self-contained  delivery 
system  with  multiple  components  and  responsibility  for  efficiently  al- 
locating resources  between  them. 

Control  for  factors  outside  the  influence  of  medical  care 

While  their  purposes  are  dissimilar,  both  in-house  monitoring  and 
policymaking  use  the  outcome  method  in  essentially  the  same  way. 
Only  one  distinction  must  be  made:  the  stringency  with  which  the  ef- 
fects on  outcomes  of  factors  outside  the  influence  of  the  medical  care 
system  are  controlled  when  outcome  data  are  analyzed.  Depending  on 
the  purpose,  some  type  of  control  is  necessary  in  order  to  conclude  that 
differences  in  outcome  are  attributable  to  differences  in  quality  of  care 
delivered,  rather  than  to  differences  in  patient-  or  disease-related  fac- 
tors that  cannot  be  altered  by  the  medical  care  system. 

Methods  of  control  must  be  far  more  stringent  when  retrospective 
monitoring  is  undertaken  for  policy-relevant  quality  assessment  studies 
than  for  in-house  studies.  This  is  necessary  because  the  major  product 
of  the  policy-relevant  study  is  detection  of  moderate  differences  in 
quality  between  system  characteristics  of  concern  and  development  of 
policies  that  will  reduce  or  eliminate  such  differences.  In  contrast,  in- 
house  assessment  is  undertaken  to  identify  major  deficiencies  in  care, 
which  the  method  itself  is  sensitive  enough  to  detect  without  rigorous 
controls. 

Approaches  to  development  of  outcome  criteria  and  standards 

Once  the  outcome  method  has  been  chosen  as  appropriate  for 
quality  assessment,  the  disease  condition  for  study  must  be  selected, 
outcome  criteria  identified,  and  standards  of  care  set.  Outcomes  are 
then  measured,  and  a  value  judgment  made  as  to  whether  care  is  of 
good  or  poor  quality  in  relation  to  the  standards  established.  There  are 
several  approaches  to  each  of  these  tasks.  Many  different  criteria  are 
available  to  guide  the  selection  of  disease  conditions,  such  as,  the  out- 
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come  criteria  selected  can  be  either  specific  concepts  (such  as  mortality 
or  functional  status)  or  a  global  concept  of  "overall  health  status";  the 
criteria  chosen  can  represent  one  or  multiple  dimensions  of  health; 
standards  can  be  implicit  or  explicit;  several  different  points  in  time 
can  be  used  for  measurement  of  the  outcome;  and  different  ap- 
proaches can  be  taken  to  evaluating  the  results  of  the  quality  assess- 
ment. Each  of  these  topics  is  dealt  with  in  turn  in  the  first  volume  of 
the  report,  both  from  a  theoretical  point  of  view  and  with  reference  to 
the  approaches  used  in  quality  of  care  studies  that  have  relied  on  the 
outcome  method  and  are  reported  in  the  literature.  (An  extensive  lit- 
erature review,  in  tabular  form,  is  appended  to  that  volume.)  Problems 
that  may  be  encountered  in  criteria  selection  and  standard  setting  with 
both  uses  of  the  outcome  method  are  chiefly  due  to  lack  of  information 
and  guidelines  in  clinical  and  health  services  research  literature,  and 
are  reviewed  in  some  detail. 
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□  □  □ 


Development  of 
short-term 
disease-specific 
outcome  measures 


Outcome  criteria  and  standards  were  developed  for  eight  condi- 
tions and  surgical  procedures:  asthma  in  children  and  adults;  breast 
mass  and  breast  cancer;  cholecystectomy;  diarrhea  and  dehydration  in 
children;  ischemic  heart  disease  (including  angina  pectoris,  myocardial 
infarction,  and  sudden  death);  osteoarthrosis;  otitis  media  in  children; 
and  tonsillectomy  with  or  without  adenoidectomy.  These  conditions 
and  procedures  represent  a  mix  of  medical  and  surgical,  inpatient  and 
outpatient,  emergency  and  nonemergency,  and  adult  and  pediatric 
problems  (see  Tables  1  and  2).  This  mix  was  chosen  in  order  to  explore 
the  feasibility  of  developing  outcome  measures  across  different  areas 
of  medical  specialty,  locations  of  care  (ambulatory  versus  inpatient), 
and  patient  age  groups.  A  major  factor  affecting  this  selection  was  the 
extent  to  which  intervention  of  medical  care  has  an  impact  on  the  nat- 
ural history  of  the  disease.  In  identifying  conditions  for  study,  those 
for  which  medical  care  has  a  considerable  impact  on  psychosocial  di- 
mensions were  included  to  test  adequately  the  utility  of  the  outcome 
method  in  assessing  quality  of  care.  The  other  major  factor  guiding 
choice  of  conditions  was  the  frequency  with  which  they  occur  in  the 
general  population.  Two  other  criteria — knowledge  of  nonmedical  fac- 
tors affecting  the  natural  history  of  the  condition,  and  availability  of 
well-defined  treatment  modalities  for  the  condition — were  given  less 
weight  in  selecting  these  eight  conditions  and  procedures.  Health  prob- 
lems (i.e.,  diarrhea,  breast  mass,  osteoarthrosis,  and  ischemic  heart  dis- 
ease) were  included  deliberately;  the  majority  of  patient  contacts  with 
medical  care  are  for  symptoms,  signs,  or  problems,  and  the  feasibility 
of  developing  outcomes  for  these  as  well  as  for  specific  diagnoses  must 
be  attempted  before  the  utility  of  the  method  is  completely  deter- 
mined. 

The  outcome  standards  developed  during  this  research  are 
explicit,  disease-specific,  and  designed  for  measurement  within  one 
year  of  the  intervention  of  medical  care.  The  criteria  to  which  these 
standards  pertain  were  drawn  from  a  cluster  of  diff  erent  dimensions  of 
health.  The  generic  categories  included  physical,  physiologic,  and 
psychosocial  outcomes  that  are  affected,  positively  or  negatively,  both 
by  the  disease  process  and  by  the  treatment  process.  These  standards 
were  developed  by  panels  of  physicians  representing  a  variety  of  medi- 
cal and  surgical  specialties,  geographic  areas,  types  of  medical  care  or- 
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ganizations,  and  both  the  academic  and  practicing  physician  com- 
munities. 

These  standards  are  designed  for  use  in  retrospective  monitoring 
of  the  quality  of  care,  particularly  when  it  is  undertaken  to  support 
policy  decisions.  Care  was  therefore  taken  to  identify  all  factors  outside 
the  influence  of  medical  care  that  might  adversely  affect  outcomes  and 
to  propose  methods  that  would  control  for  this  effect  when  analyzing 
data  on  outcome  and  comparing  different  patient  groups  or  delivery 
systems.  The  standards  are  also  appropriate  for  in-house  peer  review 
systems;  when  they  are  applied  for  that  purpose,  the  control  tech- 
niques will  not  have  to  be  as  rigorous  as  those  proposed.  Table  3  pre- 
sents the  top-priority  outcome  criterion  and  standard  for  each  of  the 
eight  topics  studied  (see  complete  standards). 

Method  of  setting  standards 

Extensive  literature  reviews  were  prepared  by  project  staff  for 
each  of  the  conditions  or  procedures  selected.  The  purpose  of  these 
reviews  was  to  provide  background  materials  necessary  to  enable  ex- 
pert panels  to  identify  criteria  and  develop  outcome  standards  for  the 
conditions  and  procedures  studied.  The  panel  meetings  were  a  focal 
point  of  the  research  design.  Every  effort  was  made  to  include  the  fol- 
lowing experts  on  each  panel:  a  board-certified  physician  who  had 
done  significant  research  on  the  condition  using  process  and/or  out- 
come measures;  a  board-certified  physician  in  private  practice  who  was 
expert  in  treating  the  condition  in  a  community  setting;  and  a  board- 
certified  physician-epidemiologist,  or  a  physician  who  had  done  signif- 
icant research  on  the  condition  from  that  perspective.  Achieving  this 
breadth  on  each  panel  was  considered  important  in  assuring  that  the 
resulting  outcome  standards  would  reflect  an  aggregate  of  opinions 
and  experience  and  would  be  generally  acceptable  to  physicians  across 
the  country. 

Panel  meetings  were  chaired  by  one  of  the  three  project  staff 
physicians,  each  of  whom  assumed  responsibility  for  two  to  four  of  the 
conditions  or  procedures  selected.  After  critiquing  the  scope  and  con- 
tent of  the  literature  reviews,  the  panels  addressed  themselves  to  five 
tasks  in  development  of  outcome  criteria  and  standards:  (1)  opera- 
tional definition  of  the  disease  condition  or  procedure  and  relevant 
patient  population;  (2)  identification  of  those  factors  outside  the  influ- 
ence of  medical  care  that  could  adversely  affect  outcomes  and  that 
must  be  controlled  for  in  analyzing  outcome  data;  (3)  identification  of 
outcome  concepts  appropriate  for  measurement  in  quality  assessment, 
in  terms  of  sensitivity  to  variations  in  the  quality  of  care  provided;  (4) 
standard  setting;  and  (5)  identification  of  the  points  in  time  at  which 
the  outcome  should  be  measured.  In  developing  standards,  the  panels 
attempted  to  estimate  on  the  basis  of  the  literature  and  their  own  ex- 
perience the  proportions  of  patients  who  might  be  expected  to  have  a 
specified  status  on  a  given  outcome  criterion  if  they  had  received  care 
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of  optimal  quality  during  the  illness  episode  or  surgical  procedure.  In 
several  cases,  a  standard  indicative  of  average  care  was  also  developed. 
"Optimal"  represents  the  professional  opinion  of  what  the  best  avail- 
able care  can  achieve  given  present-day  knowledge  in  medical,  techno- 
logical, and  art-of-care  areas;  "average"  suggests  the  typical  care  deliv- 
ered by  the  average  physician  or  community  hospital. 

The  panels  recognized  that  while  certain  factors  outside  the  influ- 
ence of  medical  care  might  adversely  affect  patient  status  on  a  given 
outcome  measure,  they  might  have  no  such  effect  on  another.  Thus,  as 
outcome  standards  were  developed,  specific  factors  that  should  be  con- 
trolled for  when  analyzing  patient  status  on  each  outcome  were  iden- 
tified. The  panels  also  discussed  techniques  by  which  the  effects  of 
these  factors  could  be  controlled  when  analyzing  outcome  information. 
For  several  conditions  (asthma,  diarrhea,  breast  cancer),  a  variety  of 
disease-specific  classification  systems  that  allow  identification  of  sub- 
groups of  populations  with  similar  prognoses  were  found  in  the  litera- 
ture. Two  problems  were  encountered  with  these  staging  systems:  (1) 
variables  used  to  assign  patients  to  stages  were  so  poorly  defined  that 
use  of  the  system  would  be  impossible,  e.g.,  many  studies  stratified  pa- 
tients in  terms  of  "mild,  moderate,  and  severe"  disease,  without  provid- 
ing further  information  to  enable  retrospective  classification  of  other 
patients  into  such  categories;  and  (2)  with  the  possible  exception  of 
tumor  staging  systems,  there  are  few  widely  accepted  staging  systems 
for  any  of  the  diseases.  For  several  other  conditions  or  procedures 
(cholecystectomy,  ischemic  heart  disease),  the  panels  proposed  use  of 
multivariate  statistical  techniques  to  control  for  factors  not  under  the 
influence  of  medical  care. 

Each  of  the  disease-specific  chapters  in  the  second  volume  of  the 
final  report  contains  summaries  of  the  biomedical  and  health  services 
research  literature  on  each  condition  or  procedure,  the  outcome 
criteria  and  standards  developed  by  the  panels,  identification  of  the 
risk  factors  to  be  controlled  for  on  each  outcome  measure,  estimates  of 
the  size  of  the  sampling  frame  necessary  to  identify  samples  of  200 
patients  with  each  condition,  and  lists  of  possible  items  that  can  be  used 
in  developing  data  collection  instruments  to  measure  outcome  stand- 
ards and  control  for  external  factors. 


□  □  □ 
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Conclusions  and 
recommendations 


Quality  assessment  methods  in  general:  conclusions 

1.  A  major  reason  for  focus  on  outcomes  in  assessing  quality  of 
care  is  the  recognition  that  use  of  structural  and/or  process  variables 
alone  may  be  invalid,  and  the  belief  that  outcome  measures  have  more 
face  validity  in  that  they  focus  directly  on  health  status.  Use  of  outcome 
measures  alone,  however,  may  also  produce  misleading  results. 

2.  Little  is  known  about  the  congruence  of  the  process  and  out- 
come methods  of  assessing  quality  of  care,  and  about  which  has  more 
validity  as  a  basis  for  corrective  actions  and  policy  decisions. 

3.  Little  is  known  about  the  comparative  reliability  of  process  or 
outcome  methods  done  independently  or  in  combination  to  assess  qual- 
ity of  care. 

4.  The  relative  feasibility,  particularly  ease  of  access  to  and  cost  of 
necessary  information,  of  both  the  process  and  outcome  methods  is  not 
known. 

Quality  assessment  methods  in  general:  recommendations 

1.  Both  the  process  and  the  outcome  approaches  require  extensive 
testing  as  to  their  respective  validities,  reliabilities,  and  feasibilities. 

2.  The  combined  process-outcome  approach  also  requires  exten- 
sive testing  as  to  its  overall  validity,  reliability,  and  feasibility. 

3.  The  respective  methods  should  be  tested  for  comparative  valid- 
ity, reliability,  and  feasibility,  to  determine  which  is  the  "best"  ap- 
proach to  use  depending  on  the  purpose  of  quality  assessment  and  the 
conditions  selected  for  study. 

Specific  conclusions  regarding  this  method 

1.  The  approach  used  during  this  research  to  develop  outcome 
criteria  and  standards  for  quality  of  care  assessment  was  a  successful 
one.  For  each  of  eight  disease  conditions  or  surgical  procedures  con- 
sidered, expert  panels  were  able  to  reach  consensus  and  to  develop 
outcome  criteria  and  standards.  These  most  basic  conclusions  are  all 
the  more  important  given  the  fact  that  most  panel  members  had  not 
been  involved  previously  in  similar  tasks,  nor  had  many  of  them  ever 
examined  the  quality  of  care  delivered  to  groups  of  patients. 

2.  A  major  reason  for  the  success  of  the  panel  meetings  was  the 
extensive  background  work  done  by  the  project  staff  to  identify  issues 
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appropriate  for  developing  outcome  measures  for  each  of  the  condi- 
tions and  procedures,  including  lengthy  literature  reviews  on  each 
topic.  Without  such  background  information,  it  is  doubtful  that  the 
panels  could  have  reached  agreement  as  readily  or  accomplished  as 
much  as  they  did  during  the  panel  meetings. 

3.  The  tasks  undertaken  by  the  panels  proved  to  be  educational 
and  stimulating  for  physicians  and  health-related  professionals.  Virtu- 
ally all  the  panel  members  were  intrigued  by  the  issues  of  quality  as- 
sessment using  outcome  information,  and  indicated  considerable  inter- 
est in  participating  in  a  second  phase  of  study  to  test  the  reliability, 
validity,  and  feasibility  of  the  outcome  standards  they  developed.  In 
addition,  most  suggested  that  the  disease-specific  chapters  would  be 
valuable  contributions  to  the  clinical  literature  and  indicated  that  these 
papers,  including  standards,  should  be  published. 

4.  The  cost  of  this  approach  to  developing  outcome  criteria  and 
standards  was  approximately  $25,000  to  $30,000  for  each  disease  con- 
dition and  surgical  procedure  studied.  These  costs  included  panel 
honoraria,  travel  expenses,  and  considerable  staff  time  spent  in  prepar- 
ing background  papers  for  panel  meetings  and  in  rewriting  them  to 
include  the  results  of  panel  deliberations.  If  one  percent  of  medical 
care  expenditures  is  to  be  used  for  quality  assurance  activities,  the  cost 
of  this  activity  represents  0.025  percent  of  such  expenditures.  If  these 
papers  can  be  used  by  many  quality  assurance  systems,  agencies,  or 
programs  as  background  information  upon  which  to  base  a  quality  as- 
surance program,  then  the  costs  seem  justifiable.  If,  however,  each  or- 
ganization doing  quality  assurance  feels  compelled  to  replicate  this 
work,  costs  could  become  excessive. 

5.  Several  problems  were  identified  during  the  literature  reviews 
and  panel  meetings  that  have  important  consequences  for  the  use  of 
outcome  information  in  quality  assessment.  First,  results  of  tests  of  the 
efficacy  of  medical  processes  and  therapies  reported  in  the  literature 
usually  represent  those  from  the  "best"  clinical  series.  Absence  of  in- 
formation as  to  what  outcomes  might  result  if  average  care  were  deliv- 
ered to  the  population  makes  choice  of  appropriate  outcome  criteria 
and  standards  problematic.  Second,  few  outcomes  (generally 
physiologic  and  physical  rather  than  psychosocial)  are  measured  in  ef- 
ficacy studies.  Absence  of  the  use  of  a  broad  range  of  outcomes  to  es- 
tablish the  efficacy  of  medical  care  process  makes  selection  of  the  "most 
important"  outcome  criteria  for  subsequent  quality  of  care  studies  dif- 
ficult. Third,  much  of  the  conventional  clinical  wisdom  upon  which 
choice  of  outcomes  and  points  in  time  at  which  to  measure  them  (time 
windows)  might  be  based  has  not  been  documented  in  the  literature, 
nor  has  the  relative  cost  and  benefit  of  monitoring  outcomes  tradition- 
ally selected  by  clinicians  been  examined.  Fourth,  neither  the  natural 
history  of  many  common  diseases  nor  the  epidemiology  of  outcome 
events  is  well  documented,  making  precise  definition  of  disease  condi- 
tions for  study  and  choice  of  "time  windows"  difficult. 
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6.  As  a  result  of  these  problems,  the  outcome  criteria  and  stand- 
ards developed  by  the  panels  are  not  yet  ready  for  use  in  ongoing  qual- 
ity assessment  or  assurance  activities.  The  criteria  and  standards,  as 
well  as  the  "time  windows"  selected  for  measurement  and  the  items 
pools  proposed  for  use  in  data  collection,  must  be  subjected  to  rigorous 
pretesting  and  subsequent  refinement  prior  to  use. 

7.  Sample  sizes  for  quality  assessments  based  on  the  outcome 
criteria  and  standards  developed  for  eight  conditions  and  procedures 
will  depend  on  several  factors;  most  important  is  the  frequency  with 
which  the  given  outcome  occurs.  Rough  estimates  suggest  that  the 
sampling  frames  necessary  to  identify  samples  of  patients  with  these 
conditions  may  preclude  outcome  assessments  focused  on  single  prac- 
tices or  hospitals,  unless  the  purpose  is  to  identify  physicians  whose 
practice  is  so  deficient  as  to  be  almost  negligent,  because  the  number  of 
patients  seen  during  a  year  will  not  be  sufficient  for  statistical  analysis. 
Instead,  data  will  be  aggregated  across  multiple  practices  or  hospitals 
to  identify  requisite  samples,  and  the  resulting  analyses  of  outcomes 
will  focus  on  groups  of  physicians,  a  region,  or  a  medical  care  area. 

8.  In  order  to  identify  patients  for  inclusion  in  quality  assessment 
studies,  implementation  of  this  outcome  method  in  any  routine  fashion 
will  require  development  of  record  files  that  are  diagnosis- 
procedure-  and/or  problem-specific  for  both  ambulatory  and  inpatient 
care.  For  conditions  that  are  generally  encountered  in  ambulatory  care 
and  are  described  by  signs  and  symptoms  rather  than  specific  diag- 
noses, uniform  symptom  codes  are  required.  The  uniform  hospital  dis- 
charge abstract  now  being  introduced  across  the  United  States  will 
prove  valuable  in  identifying  samples  of  patients  for  those  diseases  and 
procedures  that  require  hospitalization. 

Specific  recommendations  for  further  research 
using  these  outcome  standards 

1.  Explicit,  disease-specific  process  criteria  and  standards  should 
be  developed  for  each  of  the  eight  conditions  and  surgical  procedures 
studied  in  this  research. 

2.  After  explicit  process  criteria  and  standards  are  established, 
data  collection  instruments  that  will  allow  assessment  of  both  process 
and  outcome  of  care  should  be  developed  for  each  condition  and  pro- 
cedure. 

3.  Following  development  of  data  collection  instruments,  a  decep- 
tive quality  assessment  study  of  process  and  outcome  of  care  for  several 
patient  groups  in  different  settings  should  be  undertaken.  Such  a  study 
would  test  the  reliability,  validity,  and  feasibility  of  process  and  out- 
come criteria  and  standards,  and  of  the  "time  windows"  selected,  and 
would  determine  the  costs  of  conducting  such  a  study.  It  is  suggested 
that  a  subset  of  three  or  four  of  the  eight  conditions  studied  in  this 
project  be  selected  for  testing  during  this  phase. 

4.  It  is  expected  that  the  descriptive  pretest  will  lead  to  refinement 
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of  criteria  and  standards  and  to  development  of  reasonably  valid  and 
reliable  data  collection  techniques.  Once  this  has  been  accomplished, 
an  experimental  study  should  be  undertaken  to  determine  whether  al- 
terations in  the  quality  of  care  actually  improve  health  outcomes. 

5.  While  the  experimental  study  proceeds,  value  judgments  must 
be  made  concerning  the  relative  worth  of  achieved  outcome  improve- 
ments to  determine  whether  the  improvements  in  health  outcomes  are 
worth  the  cost.  These  value  judgments  could  be  obtained  using  the 
"willingness-to-pay"  approach  and  should  include  the  judgments  of 
consumers  and  of  patients  who  have  had  the  particular  disease  or  sur- 
gical procedure. 

6.  Research  programs,  as  outlined  in  the  preceding  five  recom- 
mendations, should  begin  on  two  conditions  annually  in  order  to  pro- 
vide the  information  necessary  to  determine  the  scope,  benefit,  and 
cost  of  quality  assessment  activities  using  this  outcome  method.  The 
research  cost  would  be  about  $250,000  per  condition.  While  this  work 
proceeds,  work  similar  to  that  completed  during  the  research  reported 
in  the  first  two  volumes  of  this  report  should  be  done  on  eight  to  ten 
new  conditions  annually,  at  a  cost  of  about  $300,000  per  year.  Thus, 
funding  of  about  $800,000  per  year  would  be  required  to  carry  out 
such  a  research  and  development  effort  whose  purpose  was  to  extend 
the  utility  of  the  outcome  method  for  quality  assurance. 

Policy  recommendations 

1.  When  studies  on  efficacy  of  medical  care  processes  are  funded, 
consideration  should  be  given  to  designing  them  so  that  the  results  are 
directly  useful  in  assessing  quality  of  care.  This  would  require  the  use 
of  a  broader  range  of  outcome  measures,  including  where  appropriate 
the  psychosocial  aspects  of  care,  and  would  require  the  determination 
of  the  benefit  of  a  procedure  or  therapy  when  used  under  average  as 
well  as  ideal  circumstances. 

2.  Basic  descriptive  studies  are  needed  to  determine  variations  in 
monitoring  of  outcomes  by  individual  physicians  for  the  sole  purpose 
of  delivering  medical  care.  These  descriptive  studies  should  be 
analyzed  carefully  and  followed  by  experimental  or  quasi-experimental 
studies  to  determine  the  health  effect  of  such  variations.  Such  informa- 
tion could  be  used  when  outcome  standards  are  developed  for  quality 
of  care  assessments. 

3.  If  the  approach  taken  during  this  research  is  to  be  followed,  it 
will  be  necessary  to  update  the  reviews  of  the  disease-specific  literature 
every  three  to  five  years,  depending  on  the  condition  and  amount  of 
relevant  biomedical  and  health  services  research  done  in  the  interven- 
ing period-  An  ongoing  mechanism  to  accomplish  this  updating  and  to 
disseminate  results  should  be  developed. 

4.  The  report  indicates  the  need  to  use  a  flexible  approach  in 
mandating  use  of  specific  outcome-oriented  approaches  in  operational 
quality  assessment  programs.  At  this  time,  it  is  probable  that  the  major 
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purpose  of  operational  quality  assessment  programs  is  the  identifica- 
tion of  those  physicians,  practicing  exceptionally  poor  medicine.  This 
task  could  be  accomplished  by  monitoring  the  process  of  care  alone  or 
perhaps,  for  surgeons,  death  rates  from  common  surgical  procedures. 
A  more  comprehensive  system  of  quality  assurance  that  will  help  all 
physicians  to  practice  better  medicine  awaits  answers  to  the  fundamen- 
tal questions  addressed  in  this  report.  What  tradeoff  does  the  public 
desire  between  quality  of  life  and  length  of  life,  if  both  cannot  be 
maximized  simultaneously  owing  to  the  nature  of  the  disease?  Who 
should  set  health  outcome  priorities:  the  public  or  the  physician?  How 
much  is  the  public  willing  to  pay  either  directly  or  indirectly  for  what 
level  of  health  benefit?  Answers  to  these  questions  would  be  easier  to 
obtain  if  information  covering  the  marginal  effect  on  health  of  optimal 
versus  average  care  were  available.  This  information  is  not  in  the  litera- 
ture, and  efforts  to  collect  the  data  and  make  these  decisions  should  be 
undertaken. 

5.  Two  other  uses  for  this  outcome  information  were  discussed  in 
this  report:  (1)  as  dependent  or  control  variables  in  studies  that  seek  to 
address  policy-relevant  questions;  and  (2)  as  techniques  by  which 
physicians  monitor  the  care  of  their  individual  patients.  Information 
used  in  these  ways  serves  a  broad  policy  perspective  or  results  in  the 
monitoring  of  day-to-day  care.  Both  uses  could  be  improved  remarka- 
bly if  the  influence  of  factors  not  under  control  of  medical  care  on 
outcomes  were  better  understood  and  if  the  complex  relationship 
among  these  factors,  the  quality  of  care,  and  the  cost  of  such  care  could 
be  determined.  This  better  understanding  is  conditional  upon  de- 
velopment of  a  research  program  addressing  such  issues. 
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Current  NCHSR  publications 


The  following  National  Center  for  Health  Services  Research  publications 
are  of  interest  to  the  health  community.  Copies  are  available  on  request 
to  NCHSR,  Office  of  Scientific  and  Technical  Information,  3700  East- 
West  Highway,  Room  7-44,  Hyattsville,  MD  20782  (tel.:  301/436-8970). 
Mail  requests  will  be  facilitated  by  enclosure  of  a  self-adhesive  mailing 
label.  Note:  Numbers  in  parentheses  are  order  numbers  for  the  National 
Technical  Information  Service,  Springfield,  VA  22161  (tel.:  703/557- 
4650). 

Research  Management 

The  Research  Management  Series  describes  programmatic  rather  than 
technical  aspects  of  the  NCHSR  research  effort.  Information  is  presented 
on  the  NCHSR  goals,  research  objectives,  and  priorities;  in  addition,  this 
series  contains  administrative  information  on  funding,  lists  of  grants  and 
contracts,  and  special  programs.  Publications  in  this  series  are  intended 
to  bring  basic  information  on  NCHSR  and  its  programs  to  research  plan- 
ners, administrators,  and  others  who  are  involved  with  the  allocation  of 
research  resources. 


(HRA)  76-3136  The  Program  in  Health 
Services  Research  (Revised  9/76) 

(HRA)  77-3158  Summary  of  Grants  and 
Contracts  Active  June  30,  1976 

(HRA)  77-3167  Emergency  Medical  Serv- 
ices Systems  Research  Projects  (Active  as 
of  June  30,  1976)  (PB  264  407) 


(HRA)  77-3179  Research  on  the  Priority 
Issues  of  the  National  Center  for  Health 
Services  Research,  Grants  and  Contracts 
Active  on  June  30,  1976 

(HRA)  77-3173  Summary  of  Recent 
Studies  in  Health  Services  Research,  CY 
1976  (in  preparation) 


Policy  Research 

The  Policy  Research  Series  describes  findings  from  the  research  pro- 
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